Arms Control and Intelligence Explosions

A number of commentators have argued that some time in the 21st century humanity will develop generally intelligent software programs at least as capable as skilled humans, whether designed ab initio or as emulations of human brains, and that such entities will launch an extremely rapid technological transformation as they design their own successors. The speed of such a “Singularity” or “intelligence explosion” would be so great that biological humans would lack time for extensive deliberation regarding or supervision of the process. Several authors have called for regulation to retard the pace of advancement in this field, allowing more time to ensure that any intelligent machines are safe and broadly beneficial, while various proponents of the Singularity hypothesis have replied that such attempts will fail because of competition between regulatory jurisdictions, sometimes making analogies to the failures of nuclear counter-proliferation efforts. This paper discusses some key considerations that distinguish the case of sapient software programs from the historical experience with nuclear weapons technology. Shulman, Carl. 2009. “Arms Control and Intelligence Explosions.” Paper presented at the 7th European Conference on Computing and Philosophy (ECAP), Bellaterra, Spain, July 2–4. This version contains minor changes. First, the rapid and competitive development of the technology poses unusually great risks of unintentional harm, harm which would affect all competitors. An arms race that lead to a trade-off of safety for speed might result in the creation of superintelligent beings indifferent to human welfare, while natural selection could result in an initially benign population evolving into extremely competitive replicators. Second, an intelligence explosion could result in extreme winner-take-all effects, as one power could lever an initial advantage to develop astronomically more capable intelligences and prevent others from duplicating its success. Such a capacity for unilateral dominance is historically unprecedented, even by the brief window of American nuclear monopoly. Third, the very hypothesized capabilities in computing, neuroscience, and artificial intelligence that could enable a Singularity would also provide powerful new means to enforce regulations and international agreements. In combination, these factors suggest that regulatory jurisdictions may find cooperative control of the development of software entities more desirable and more practically feasible than historical nuclear arms control efforts.